COMPACTNESS OF HANKEL OPERATORS AND ANALYTIC DISCS 
IN THE BOUNDARY OF PSEUDOCONVEX DOMAINS 
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Abstract. Using several complex variables techniques, we investigate the interplay 
between the geometry of the boundary and compactness of Hankel operators. Let (3 
. be a function smooth up to the boundary on a smooth bounded pseudoconvex domain 

Q C C". We show that, if il is convex or the Levi form of the boundary of O is of rank at 
least n — 2, then compactness of the Hankel operator Hp implies that [3 is holomorphic 
"along" analytic discs in the boundary. Furthermore, when is convex in C 2 we show 
that the condition on (3 is necessary and sufficient for compactness of Hp. 



> 

u 



1. Introduction 



Hankel operators form an important class of operators on spaces of holomorphic func- 
tions. Initially there were two descriptions of Hankel operators, one considered it as an 
CSJ ■ operator on the one-sided sequence space I 2 into itself, and the other as an operator from 

^ ■ the Hardy space H 2 of the unit disk into its orthogonal complement in L 2 . These op- 

erators are closely connected to problems in approximation theory as shown by now the 
famous work of Nehari |Neh57] on one hand, and Adamjan, Arov and Krein [AAK71] 
on the other. These operators also have a close connection to Toeplitz operators, and 
the commutators of projections and multiplication operators on L 2 . More about Hankel 
operators and related topics can be found [Pel03] 

Let Q be a bounded domain in C n and dV denote the Lebesgue volume measure. The 
Bergman space A 2 (Q) is the closed subspace of L 2 (Q) consisting of holomorphic functions 
on Q. The Bergman projection P is the orthogonal projection from L 2 (Q) onto A 2 (Q) 
and can be written explicitly as Pf(z) = f n K(z,w)f(w)dV(w), where K(z,w) is the 
Bergman kernel of Q. For (3 G L 2 (Q) we can define the Hankel operator Hp from A 2 (Q) 
into L 2 {VL) by Hp(g) = (Id — P)((3g). In general, Hp is only densely defined on A 2 (Q). 
When Q is a bounded pseudoconvex domain, Kohn's formula P = Id — d Nd (N is the 
(bounded) inverse of complex Laplacian, dd +d d, and d is the Hilbert space adjoint of d 
on the square integrable (0, l)-forms on Q) implies that H p (f) = d Nd(J3f) = d N(fdf3) 
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for / G A 2 (Q) and (3 G C 1 (fi). This will be the main tool in this paper as it will allow 
us to use several complex variables techniques to study Hankel operators. We refer the 
reader to |CS01j for more information on the <9-Neumann operator. 

The study of the size estimates of Hankel operators on Bergman spaces has inspired 
a lot of work in the last 20 years. The first result in the study of boundedness and 
compactness of Hankel operators was done by Axler [Axl86j on the Bergman space of 
the open unit disk A. He showed that, for (3 holomorphic on A, H-s is bounded if and 
only if (3 is in the Bloch space, and H-a is compact if and only if (3 is in the little Bloch 
space. In the case of a general symbol, Zhu |Zhu87] showed the connection between size 
estimates of a Hankel operator and the mean oscillation of the symbol in the Bergman 
metric. In [BBCZ90] , Bekolle, Berger, Coburn and Zhu studied the same problem in the 
setting of bounded symmetric domains in C™ with the restriction that Hp and H-q are 
simultaneously bounded and compact with (3 G L 2 (Q). Stroethoff and Zheng |Str90at 
IZhe89] independently gave a characterization for compactness of Hankel operators with 
bounded symbols on A. Later Stroethoff |Str90bj generalized these results to the case 
of the open unit ball and polydisc in C n . Luecking |Lue92j gave different criteria for 
boundedness and compactness of Hp on A P (Q) with 1 < p < oo. Peloso [Pel94j extended 
Axler's result to Bergman spaces on smooth bounded strongly pseudoconvex domains. 
For the same domains, Li [Li94j characterized bounded and compact Hankel operators 
Hp with symbols (3 G L 2 (Q). Beatrous and Li [BL93] obtained related results for the 
commutators of multiplication operators and P on more general domains, that include 
smooth bounded strongly pseudoconvex domains. 

The novelty of our approach is that we put an emphasis on the interplay between the 
geometry of the domain and the symbols of Hankel operators. Although, our symbols are 
more restricted the domains we consider are much more general and allow rich geometric 
structures. 

In several complex variables, compactness of the <9-Neumann operator has been an 
active research area for the last couple of decades. We refer the reader to a very nice survey 
|FS01] for more information about compactness of the d- Neumann operator. Compactness 
of the canonical solution operators for d on the unit disk has been discussed in |Has01j . 
where it was in fact shown that this operator restricted to (0, l)-forms with holomorphic 
coefficients is a Hilbert-Schmidt operator. Fu and Straube [FS98j showed that presence 
of analytic discs in the boundary of a bounded convex domain in C n is equivalent to 
the non-compactness of the <9-Neumann operator. The second author and Straube |§S06 
used their techniques to prove that analytic discs are obstructions for compactness of the 



COMPACTNESS OF HANKEL OPERATORS 



3 



<9-Neumann operator on smooth bounded pseudoconvex domains in C™ whose Levi form 
has maximal rank. In C 2 their result reduces to a folklore result of Catlin |FS98] . 

Given Kohn's formula it is natural to expect a strong relationship between Hankel 
operators and the <9-Neumann operator. The following fact confirms this expectation. 
Compactness of the <9-Neumann operator implies compactness of Hankel operators with 
symbols are smooth on the closure [FSOlj . Actually, the statement in [FSOlj requires the 
symbol to have bounded first order derivatives. But any symbol that is continuous on the 
closure can be approximated uniformly by symbols that are smooth on the closure of the 
domain. Hence the resulting Hankel operators converge in norm preserving compactness. 
In this paper we show that the theory for compactness of Hankel operators is somewhat 
parallel to the theory of compactness of the <9-Neumann operator in terms of analytic 
structures in the boundary. Previous work in this direction was done by Knirsch and 
Schneider [KS07j . 

Throughout the paper bfl denotes the boundary of Q. Our first result concerns smooth 
bounded pseudoconvex domains in C n . 

Theorem 1. Let Q be a smooth bounded pseudoconvex domain in C n for n > 2 and 
(3 G C°°(Ji). Assume that the Levi form of bQ is of rank at least n — 2. // Hp is compact 
on A 2 (Q), then f3 o / is holomorphic for any holomorphic function f : A — > bfl. 

Remark 1. We note that the statement u (3 o / is holomorphic" can be interpreted as 
meaning that (3 is holomorphic "along" M = /(A). However it may not be holomorphic 
in the transversal directions. 

Remark 2. One can check that the proof of Theorem [1] shows that compactness of Hp on 
A 2 (Q) for (3 G C°°(Q) still implies that /3o f is holomorphic for any holomorphic function 
/ : A — > bQ when Q satisfies the following property: If the Levi form of bQ is of rank k 
for 0<k<n— l&tp, then there exists an-fc-1 dimensional complex manifold in bfl 
through p. 

Since in C 2 the Levi form has only one eigenvalue the condition on the Levi form in 
Theorem [T] is always satisfied. Therefore, for n = 2 we have the following corollary. 

Corollary 1. Let Q be a smooth bounded pseudoconvex domain in C 2 and f3 G C°°(Q). If 
Hp is compact on A 2 (Q) then (3 o f is holomorphic for any holomorphic function f : A — ► 

bn. 

For convex domains in C n we prove the same result without any restriction on the Levi 
form. 
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Theorem 2. Let Q be a smooth bounded convex domain in C n forn > 2 and (3 G C°°(Q). 
Assume that Hp is compact on A 2 (Q). Then (3 o / is holomorphic for any holomorphic 
function f : A — > 6Q. 

In the following theorem we show that, when Q is convex in C 2 , the converse of Theorem 
[T]is true. 

Theorem 3. Let Q be a smooth bounded convex domain in C 2 and (3 G C°°(Q). If (3 o f 
is holomorphic for any holomorphic f : A — > btt, then Hp is compact. 

Combining Corollary [T] (or Theorem [2]) and Theorem [3] we get a necessary and sufficient 
condition for compactness of Hp for convex domains in C 2 . 

Corollary 2. Let Q be a smooth bounded convex domain in C 2 and (3 G C°°(Q). Then 
Hp is compact if and only if (3 o / is holomorphic for any holomorphic f : A — > bfl. 

Remark 3. We note that [Mat97] constructed a smooth bounded pseudoconvex complete 
Hartogs domain Q in C 2 that has no analytic disk in its boundary, yet it does not have 
a compact <9-Neumann operator. It would be interesting to know whether there exists a 
symbol (3 G C°°(f2) such that the Hankel operator Hp is not compact on A 2 (Q). 

Remark 4. We would like to take this opportunity to point out an inaccuracy. Knirsch 
and Schneider [KS07t Proposition 1] claim that if there is an affine disk in the boundary 
of a bounded convex domain in C n , then the Hankel operator H 2 m is not compact for 
i = 1,2, ... ,n and any positive integer m where Zi is the ith coordinate function. They 
correctly prove the result when the disk lies in zi-coordinate and claim that the proof for 
i — 2, 3, . . . , n is similar. However, Theorem [3] implies that if is a smooth bounded con- 
vex domain in C 2 and the set of weakly pseudoconvex points form a disc in zi-coordinate, 
then H S2 is compact. 

Remark 5. For simplicity we assume that the domains have C°°-smooth boundary and 
the symbols are smooth up to the boundary. However, one can check that the proofs work 
under weaker but reasonable smoothness assumptions. 

Remark 6. Recently, Celik and Straube |CS] studied compactness multipliers for the d- 
Neumann problem (we refer the reader to |CS] for the definition and some properties 
of compactness multipliers of the <9-Neumann problem). This notion is related to that 
of a symbol of a compact Hankel operator, but there are differences. First of all, the 
9-Neumann operator iV is applied to square integrable forms and compactness multipliers 
are applied after N. In case of Hankel operators, however, one can think of the (0, l)-form 
d(3 as acting as a "pre-multiplier" (acting before the canonical solution operator d N) on 
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the Bergman space which is more rigid than the space of L 2 forms. Secondly, Celik 
and Straube proved that on a bounded convex domain, a function that is continuous 
on the closure is a compactness multiplier if and only if the function vanishes on all 
the (nontrivial) analytic discs in the boundary. One can show that such symbols produce 
compact Hankel operators. However, for smooth bounded convex domains in C 2 , a symbol 
smooth on the closure produces a compact Hankel operator if and only if the symbol 
is holomorphic along (see Remark [I]) analytic discs in boundary. (That is, the complex 
tangential component of the pre- multiplier on any analytic disc in the boundary vanishes). 
In general, these connections are not well understood. For example, the following question 
is still open: 

Question 1. Assume that Q is a smooth bounded pseudoconvex domain in C n and (3 G 
C(Q) is a compactness multiplier for the <9-Neumann operator on L 2 ^(fi). Is Hp compact 
on the Bergman space on Q? 

2. Proof of Theorem rjj and Theorem [2] 

Let A = Ai denote the unit open disc in C, A r denote the disc in C centered at the 
origin with radius r, and A^ denote the polydisc in C k of multiradius (r, • • • , r). We will 
be using Hankel operators defined on different domains. So to be more precise, let H9 
denote the Hankel operator on Q with symbol and Ru be the restriction operator onto 
U. Furthermore, the Bergman projection on U will be denoted by Pjj. First we will start 
with a proposition that will allow us to "localize" the proofs. 

In the proofs below we will use generalized constants. That is A < B will mean that 
there exists a constant c > that is independent of the quantities of interest such that 
A < cB. At each step the constant c may change but it will stay independent of the 
quantities of interest. 

Proposition 1. Let Q be a bounded pseudoconvex domain in C n and <p G L°°(Q). Then 

i) If H9 is compact on A 2 (Q) then for every p G bQ and U an open neighborhood of 
p such that U HQ is a domain, H^^ n u)Runn i s compact on A 2 (fl). 

ii) If for every p G bfl there exists an open neighborhood U of p such that U C\Q is a 
domain, and H^^,^R(j n Q is compact on A 2 (Q), then is compact on A 2 (Q). 

Proof. Let us prove i) first. For / G A 2 (Q) we have 

(Id UnU -P UnQ )R UnQ H$(f) = {Id vm -Puna)Runa{<t>f-Pa{<t>f)) 

= H^ nW R Unn (f) + PunnRunnPnW) - RumPn(<f>f) 
_ TjUnu p / f\ 
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In the last equality we used the fact that PunnRunnPn — RunnPn on L 2 (Q). Hence 
{Idunn ~ Punn)RunnH^(f) = H^ nn ^Runn{f)- 

Therefore, if is compact, then H^^^Runn is also compact. 

To prove ii) let us choose {pi, . . . ,p m } C bfl and open sets U\, . . . , U m such that 

i) Uj is a neighborhood of pj and Uj D Q is a domain for j = 1, . . . , m, 

ii) btt c Uf =1 Uj, 

iii) Sj = ^Rujna is compact for j = 1, . . . , m. 

Let ?7 = ^? So = H^, and {xj '■ j = 0, . . . ,m} be a C°°-smooth partition of unity 
subject to {Uj : j = 0, . . . , m}. Then for / G A 2 (fi) 

3=0 / 3=0 3=0 

m m 

= £(^)^(/) + £x;(W 

3=0 3=0 
m 

3=0 

Hence, since d (Sj=o XjSjif)) and {^4>)f are enclosed we conclude that J2j =0 (dXj)Sj(f) 
is enclosed. Let 

m m 

(1) S = J2xA-d*N^(d X] )S r 

3=0 3=0 

We write XoSo(f) as Xo^f ~~ XoPu(<f>f) and choose a bounded sequence {fj} in A 2 (Q). 
Let .fT be a compact set in Q that contains a neighborhood of the support of xo- Cauchy 
integral formula and Montel's theorem imply that {fj} and {Pn{4>fj)} have uniformly con- 
vergent subsequences on K. Then {xo^fj} and {xoPn(4>fj)} have convergent subsequences 
in L 2 (Q). That is, the operator XoSo is compact on A 2 (Q). Similarly, (dxo)So is compact 
as well. We remind the reader that we assumed that Sj is compact for j — 1, . . . , m and 
d*N n is continuous on bounded pseudoconvex domains. Therefore, ([1]) implies that S is 
a compact operator and dS(f) = {d<f))f. To get the Hankel operator we project onto the 
complement of A 2 (Q). Hence using = (Ida — Pn)S we conclude that is compact 
on A 2 (Q). □ 

Lemma 1. Let fix and Q 2 be two bounded pseudoconvex domains in C n , <p G C°°(fi2)j 
and F : Oi — > Q 2 be a biholomorphism that has a smooth extension up to the boundary. 
Assume that if? 2 is compact on A 2 (Vt 2 ). Then H^ F is compact on A 2 (Q,i). 
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Proof. Let g G A 2 (Q 1 ),f = g o F^ 1 , it = d*N n2 d(f)f, and w = u o F = F*(u). Then 
/G A 2 (fi 2 ), M = F^ 2 (/), and 

Sw; = dF*(u) = F*(du) = F*{fB<f>) = (/ o F)d(0 o F). 

So <9(w o F) = (/ o F)d(4> o F) on f2i and 9 N Ul d(u o F) is the canonical solution for 
dw = (fo F)d((f) o F) on Then 

#?oV(<?) = H$ F U ° F ) = d*N^d(u o F) = d* N Ql d(F*H^ 2 ((F~ 1 )*(g))). 

Therefore, H^ F is a composition of H^ 2 with continuous operators d*N Ul d,F*, and 
(F^ 1 )*. Then since H^ 2 is assumed to be compact on A 2 (Q 2 ) we conclude that H^ F is 
compact on A 2 (Qi). □ 

Let 0^(2) be the function defined on Q that measures the (minimal) distance from z 
to bQ. The Bergman kernel function of Q satisfies the following relation on the diagonal 
of Q x Q 

K n (z,z) = snp{\f(z)\ 2 : / G A 2 (Q), \\f\\ L2(n) < 1}. 

The following proposition appeared in [Fu94] for general pseudoconvex domains in C™ 
and in |§ah06 in the following form. 

Proposition 2. Let Q be a bounded pseudoconvex domain in C n with C 2 -boundary near 
p G bQ. If the Levi form is of rank k at p, then there exist a constant C > and a 
neighborhood U of p such that 

C 

{d bn (z)) k+2 

Proof of TheoremUi We will prove a stronger result. The proof will go along the lines of 
the proof of Theorem 1 in §S06| and the proof of (1) (2) in [ FS98] with some additional 



work. The same strategy has appeared in [Cat81l IDP811 |§ah06 . Let us assume that 

i. Q is a smooth bounded pseudoconvex domain in C n and p G bQ, 

ii. the Levi form of bQ is of rank k at p through which there exists an-fc-1 
dimensional complex manifold in bQ, 

iii. there exists non-constant holomorphic mapping / : A 71 ^- 1 — > bQ and g G A such 
that f(q) = p, Df(q) is full rank (Df is the Jacobian of /), and d(f3 o f)(q) ^ 0, 

iv. Hp is compact. 

Lemma 1 in |§S06| implies that there exist a neighborhood V of p and a local holomorphic 
change of coordinates G on V so that G(p) = 0, positive y n -axis is the outward normal 
direction to the boundary of Qi = G(V fl Q) at every point of M = {z G A n : z n _fc = 
• ■ • = z n = 0} C bQi. 
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Let z = (z', z") where z' = {z\, . . . , z n _k-i) and z" = (z n _k, ■ ■ ■ , z n ). We define L to 
be the k + 1 (complex) dimensional slice of Q\ that passes through the origin and is 
orthogonal to M. That is, L = {z" G C fc+1 : (0, z") G So L is strongly pseudoconvex 
at the origin when k > 1 and is a domain in C when k = 0. Then there exists < A < f 
such that Mi x L\ C fii, where L\ is a ball in C fc+1 centered at (0, ... , 0, —A) with radius 
A and M\ = \M. For every j we choose pj = (0, . . . , 0, G Mi x L\. We take the 

liberty to abuse the notation and consider pj = (0, . . . , 0, — 1/j) G L\. Now we define 
q. = G- 1 {pj) G V n f2 and 

^, = K n (z, qj ) 

One can check that {/j} is a bounded sequence of square integrable functions on Q that 
converges to zero locally uniformly. Let us define <x,- = fj o G" 1 and fi\ — ft o Then 
i) in Proposition CD implies that if^^/mi^ynn is compact. In turn, Lemma [U implies that 
is compact. Hence {H^(oij)} has a convergent subsequence. The strategy for the 
rest of the proof will be to prove that {Hp^aj)} has no convergent subsequence. Hence, 
getting a contradiction. 



Since d{f3 o f)(q) ^ without loss of generality we may assume that 



> at the 

origin. Then there exist < r < 1 and a smooth function < x < 1 on real numbers 
such that 



i. 
ii. 
iii. 



A n-k-i c Mi) 

X(t) = lfor \t\<r/2, X (t) 



for |t| > 3r/5, 



> on AI 



Then C = fi zl t <3r / 4 x{\ z i\)dV(zi) > 0. Let us define 7 on Qi so that 

7(«)^^ = xW)--x(W)- 

and (., .) denote the standard pointwise inner product on forms in C. Furthermore, let 
z = (zi, w) where w = (z 2 , ■ ■ ■ , z n ) and a G A 2 (Qi). Then using the mean value property 
for a holomorphic function a and for fixed w G A™^* so that A r x {w} c x we get 

Ca(0,w) = / x(\ z i\) a (zi,w)dV(z 1 ) 

J\z!\<3r/4 

7(21, w) — a(zi, w)dV{z 1 ) 

zi|<3r/4 OZ\ 
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On the other hand, 



7(zi, w) — a{z 1 ,w)dV{z 1 ) 

|zi|<3r/4 VZ\ 



(cud /3i, jdzi)dV(zi] 



|2l|<3r/4 



(dd*N ni (adp 1 ),jdz 1 )dV(z 1 ] 



2l|<3r/4 



Zi|<3r/4 uz l 



z 1 \<3r/i 



H%{a)^-dV{ Zl ). 



Therefore, we have 



a(0, w)\ 


s 1 


7 i 






V|*l|<3r/4 



1/2 



If we square both sides we get 
\a(0,w)\ 2 ; 



2i|<3r/4 



Since |a(0, u>)| 2 is subharmonic when we integrate over (z 2 , ■ ■ ■ , z n -k-\) G 2 ; we get 



(2) 



/A|2 



l«(o,/)l 



< 



3r/4 



The above inequality applied to ay implies that (X/Ilj G L 2 (Li). Now we use the repro- 
ducing property of K Ll on aj]^ to get 

a i(Pi) = / K L 1 (Pj,z)a j \ Ll (z)dV(z). 

JLi 

Cauchy-Schwartz inequality implies that |aj(pj)| < ||«j|£i|U 2 (Li)||^Li(Pj, 0IU 2 (£i)- On 
the other hand {{K^ipj, -)IU 2 (£i) = V^ L i (PjiPj)- S° we nave 

Wj{Pj)\ 



ICtilLiIlL 2 ^!) 



> 



KLi(Pj,Pj) 



'KhAPjiPj) 

Since Li is a ball in C fe+1 and the rank of the Levi form for Q (and hence for is at least 
k, the asymptotics of the Bergman kernel on balls and Proposition [2] imply the following 
inequalities: 

(d bLl (p 3 )) k+2 ~ ~ (d bLl ( Pj ))^ 



< 
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We note that pj and qj are related by a diffeomorphism. So for large enough j d^ipj) = 
dbL L (Pj) and they are comparable to dbn(Qj). Therefore, there exists £ > such that 
£ < ||<x,-|li ||l 2 (Li) f° r all j- Since {<x,} converges to locally uniformly this implies that 
{cKjIi^} has no convergent subsequence in L 2 (Li). Also ([2]) applied to <x,- — a k implies that 

HajUi - oi k \ Ll \\ L 2 {Ll) < \\H^(aj - a k )\\ L 2 {ni) . 

Hence {H^(aj)} has no convergent subsequence in L 2 (Qi). Therefore, we have reached 
a contradiction completing the first proof of Theorem [TJ □ 

A weaker version of the following lemma appeared in [FS98J. 

Lemma 2. Let Q be a convex domain in C n and f : A — > bQ be a non-constant holo- 
morphic map. Then the convex hull of /(A) is an affine analytic variety contained in 
bQ. 

Proof. Let K be the convex hull of /(A) in C n . First we will show that K is an analytic 
affine variety. By definition K is an affine set in C n . Let F(z, w, t) = tf(z) + (1 — t)f(w) 
for (z, w) G A 2 and < t < 1. One can check that 

K = {F{z, w, t) : (z, w, t) e A 2 x (0, 1)}. 

If K is open in C" then we are done. Otherwise, there exists p G K which is a boundary 
point and, by convexity, there exists (z , Wq, t ) G A 2 x (0,1) such that after possible 
rotation and translation p = F(zo,wo,to) is the origin and K C {x n < 0}. Let us define 
g = Re{z n o F) : A 2 x (0, 1) -> R. Then g{z , w ,t ) = and #(A 2 x (0, 1)) C {x G E : 
x < 0}. Maximum principle applied to the harmonic function g implies that g = 0. Hence 
K C {x n = 0}. Since / is holomorphic, /' must stay in the complex tangent subspace of 
{x n = 0}. That is, 

f d d ] 

(3) f(p) C span j — , . . . , — — | for every p G A. 

Now it is easy to see that ([3]) implies that K C {z n = 0}. So we have demonstrated 
that if -fT is not an n dimensional analytic affine variety then it is contained in an n — 1 
dimensional analytic affine variety. We use the above argument multiple times if necessary 
to show that K is open in an analytic affine variety. Hence K is an analytic affine variety. 

Now we will show that K is contained in bQ. Since K and Q are convex after some 
possible rotation and translation, we can assume that /(0) is the origin and /(A) C Q C 
{x n < 0}. Since 7^ /(A) C K D bQ the set K is not an open set in C n . Then, as in the 
above paragraph, one can show that K C {x n = 0} fl Q C 60. This completes the proof 
of the lemma. □ 
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Proof of Theorem [H The proof will be very similar to the first part and the proof of 
(1) =>- (2) in [FS98J. So we will just sketch the proof and point out differences. Let 
us assume that H^f is compact and that there exists a nonconstant holomorphic map 
/ : A — > bQ. We can choose p G A such that \d{(3 o f){p)\ > 0. By applying translation 
and rotation, if necessary, we may assume that f{p) = 0, f'{p) = (1, 0, . . . , 0), and positive 
x n -axis is the outward normal for bfl at 0. Using Lemma [2] with scaling, if necessary, we 
may assume that {(z, 0, . . . , 0) G C n : \z\ < 1} C bQ and 1 > o. We define 

L = {{z 2 , ...,z n )e C"- 1 : (0, z 2 , ... , z n ) G ft}, 

Pj = (0, . . . , G L, and fAz) = * l{z ^ ] . Using the proof of (1) (2) in [FS98] 

\/^L(qj,qj) 

one can easily prove that {fj} is a bounded sequence in A 2 (L) such that {-Ral (/?)}, 
the restricted sequence of {fj} to XL, has no convergent subsequence in A 2 (XL) for any 
< A < 1. Then for each j we extend fj to Q using Ohsawa-Takegoshi theorem [OT87] 
to get a bounded sequence {aj} on A 2 (Q). Using similar arguments as in the proof of 
Theorem [1] and the fact that A1/2 x |L C f2 (this follows from convexity of fl) one can 
show that 

Wfj - fk\\ L \\L) ^ \\H%{<Xj - a k )\\ L 2 (n) . 
This contradicts the assumption that Hp is compact. □ 

3. Proof of Theorem [3] 

We refer the reader to [D'A02t Proposition V.2.3] for a proof of the following standard 
lemma. 

Lemma 3. Let T : X — > Y be a linear operator between two Hilbert spaces X andY . Then 
T is compact if and only if for every e > there exist a compact operator K e : X — > Y 
and C e > so that 

\\T(h)\\ Y < e\\h\\ x + C e \\K e (h)\\ Y for h G X. 

Proof of Theorem^ Let K denote the closure of the union of all analytic discs in bfl. Let 
us choose a defining function p for Q so that || Vp|| = 1. Let (3 = f3\ + z/3 2 , 

dp d dp d ^ dp d dp d 



v 



_l_ an( i T = 

f)ir ■ rim ■ fin ■ r)ni ■ ' < J 



^dxjdxj dyjdyj' dxj d Vj d yj dxj' 

For sufficiently small e and £ G bQ, let us define 

M + ^(0) = A(0 + eT{{3 2 )(£) and + ex/(0) = &(0 - eT(A)(0- 

Then (3 = fli + i/3 2 is a smooth function in a neighborhood of fefi and it is equal to (3 
on the boundary of Q. Let us extend (3 as a smooth function on f2 and still call it (3. 
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One can check that iy + iT)((3) = on bfl. That is, in some sense (3 is holomorphic 
along complex normal direction on the boundary. Let us define (3 = (3 — (3 on Vt. Then 
(3 and (3 are smooth functions on Vt such that (3 = on bVt and (3 is holomorphic on K. 
Montel's theorem together with the fact that (3 can be approximated by smooth functions 
supported away from the boundary imply that H~ is compact on A 2 (Q). In the rest of 
the proof we will show that H~ is compact on A 2 (Q). Let {ipj} be a sequence in C^^(£l) 
such that ipj = in a neighborhood of K for all j and ipj converges to 8(3 uniformly on Q. 
On the boundary, ijjjS are supported on sets that satisfy property (P) (see |FS98j when 
Q is convex). 

In the following calculation (., -)l 2 (q,) denotes the L 2 inner product on Q and N = N n . 
Now we will show that H~ is compact. Let g G A 2 (Q). Then we have 

(d*N(gdp),d*N(gdp)) L i [n) = (N (gdp) , gd?) LHn] 

= (N{gdp),90-^))^(i) + (N{gW), 9^)^n)- 

Let us fix ipj. We choose ip G C°°(Q) such that 0<^<l,^ = lon the support of ijjj 
and ip is supported away from K. Then for g G A 2 (Q) we have 

(4) \(N(gdP),g^) LHn) \ = \&N(gdp), g^) LHQ) \ < \\i>N{gdp)\\ LHQ) \\g\\ LHQ) . 

Let us choose finitely many balls B\, ... , B m and <pj G C^(Bj) for j = 0,1, ... ,m (we 
take Bo = Q here) such that 

i- EJlo <t>j = ^ on fi, 

ii. Q fl Bj is a domain for j = 1, 2, . . . , m, 

iii. U^l 1 5j covers the closure of the set {z G 6f2 : ^(z) 7^ 0}, 

iv. Q H 5j has a compact <9-Neumann operator for j = 1,2, ... ,m. 

We note that multiplication with smooth functions preserves the domain of d* and the 
<9-Neumann operator is compact on Bj fl Q for j = 1, . . . , m. Compactness of N implies 
the so-called compactness estimates (see for example [FS01] ). Let W~ 1 (fl) denote the 
Sobolev -1 norm for functions and forms. Then for every e > there exists C e > such 
that for h G L 2 ^ (Q) in the domains of d and d* we have 

m 

j=0 
m 

~ S e (P(M)IU 2 (n) + \\d*((f>jh)\\mn)) + C^l Mil w-*(n) 

5=0 

< £(11^11x2(0) + ||9*/i|| L 2 (Q) + ||/i|| L 2 (n) ) + C £ ||/i|| W -l (n ). 
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In the calculations above we used interior ellipticity for j = and the fact that multi- 
plication by a smooth function is a continuous operator on Sobolev spaces. Now if we 
replace h by Nh and use the fact that ||iV7i|| L 2 (n) + ||<97V7i|| L 2 (n) + \\d* Nh\\ L 2 {n) < \\h\\ L 2 {n) 
we get 

\\ijjNh\\ L 2 {n) < e\\h\\ L 2 {n) + C E \\Nh\\ w -i(ty for h G Lf 01) (Q). 

Then Lemma [3] implies that ipN is compact on L 2 Then using the small constant- 

large constant inequality (2ab < ea 2 + b 2 /e) combined with the inequality above and (jlj) 
we get that for any e > there exists C e > such that 

(5) \(N{00),9l>j)v>w\ < 49\\h m + Ce\\N(gdmw-, m for g G A 2 (n). 

Since tpj converges to d/3 uniformly on Q for every e > there exists ipj such that 
\(N(gd/3), g{d(3 — ipj))L 2 {n)\ < £ lblli2(Q)- Furthermore, the last inequality together with 
([5]) imply that there exists C £ > such that 

\\d*N(gdmh(n) = \\Hj!(9)\\hw Z e\\g\\% m + C e ||iV(^)||^ 1(n) for g G A 2 (fi). 

The above inequality combined with Lemma [3] and the fact that W~ l (Q) imbeds com- 
pactly into L 2 (Q) imply that H~ is compact on A 2 (Q). Therefore, is compact. □ 
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